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Medical  Modeling  and  Simulation  represents  a  relatively  new  venue  for  the  Modeling 
and  Simulation  (M&S)  community.  While  several  studies  have  been  conducted  for 
training  efficacy  as  a  measure  of  effectiveness,  no  consensus  has  been  reached  as  to  the 
appropriate  methodologies  to  provide  test  and  evaluation  strategies  for  verification, 
validation,  and  accreditation.  The  United  States  Army  Research,  Development  and 
Engineering  Command  (RDECOM)  developed  a  new  technology  for  medical  simulation 
called  the  Stand  Alone  Patient  Simulator  (SAPS).  This  development  represents  disruptive 
technology  that,  once  fielded,  will  significantly  increase  combat  medical  readiness  .  The 
SAPS  has  qualified  for  accelerated  fielding  under  the  DoD  Challenge  Program. 
Consequently,  RDECOM  has  contracted  to  conduct  an  independent  test  and  evaluation  at 
joint  selected  military  locations  under  that  program.  The  ejfort  will  be  a  precedent  setting 
event  for  the  world  of  medical  modeling  and  simulation.  This  test  will  represent  the  first 
time  a  mannequin  based  patient  simulator  has  gone  through  the  Verification,  Validation, 
and  Accreditation  (V,V&A)  process.  It  will  also  represent  the  full  application  of  the 
system  engineering  process  to  a  test  and  evaluation  construct. 
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The  Defense  Science  Board  (DSB)  established  a  task  force  on  developmental  testing  in 
2007.  This  study  completed  in  2008  made  some  significant  observations.  “The  DSB’s 
findings  concluded  that  systemic  changes  to  acquisition  processes  and  a  lack  of  a 
disciplined  systems  engineering  process  have  resulted  in  the  high  failure  rates  in 
suitability”  (DiPetto,  2008).  To  often,  research  and  development  efforts  do  not  lead  to 
viable  production  and  fielding  programs  due  to  transitional  documentation  deficiencies 
and  their  associated  corrective  action  strategies.  A  maturing  technology  requires 
complete  documentation  and  corrective  action  management  from  concept  through 
prototype  and  first  article  development  into  acquisition  for  production  and  fielding.  As 
part  of  the  research  and  development  effort,  these  transition  packages  must  be  completed 
in  order  to  facilitate  the  movement  necessary  to  bridge  the  research  and  development 
phase  into  acquisition  and  fielding  in  a  controlled  process  environment.  In  addition, 
acquisition  and  fielding  funding  cannot  be  readily  obtained  without  these  transition 
packages.  In  the  worst  case,  life  cycle  funding  can  exceed  developmental  funding  for 
reliability  failures.  All  these  issues  are  factored  against  tremendous  pressure  to  get 
systems  into  war  fighter’s  hands.  Given  the  current  OPTEMPO  in  asymmetric  warfare, 
deploying  forces  need  solutions  now  and  cannot  wait  for  traditional  waterfall 
developmental  processes.  This  dilemma  can  be  resolved  by  using  effective  and  efficient 
system  engineering  processes  for  requirements  traceability  to  educational  tasks,  and 
detailed  functional,  component,  and  data  flow  descriptions  within  the  conduct  of 
developmental  testing. 

Medical  Simulation 

Medical  Modeling  and  Simulation  represents  a  relatively  new  venue  for  the  Modeling 
and  Simulation  community,  particularly  for  test  and  evaluation.  Currently,  the  issue  of 
effective  test  and  evaluation  matrixes  for  medical  simulation  has  generated  much 
discussion  within  the  medical  community  (van  Meurs,  et  al,  1997,  DeVita,  et  al,  2004, 
Bradley,  2006).  While  several  studies  have  been  conducted  for  training  efficacy  as  a 
measure  of  effectiveness,  no  consensus  has  been  reached  as  to  the  appropriate 
methodologies  to  provide  test  and  evaluation  strategies  for  verification,  validation,  and 
accreditation  (Gordon,  et  al,  2003, Gallagher,  et  al,  2004,  Meier,  et  al,  2005).  In  addition, 
the  suitability  of  one  particular  simulation  (CD  ROM)  over  another  (patient  simulator)  is 
just  now  being  considered.  (Johnson,  et  al,  2008)  However,  there  is  a  general  consensus 
that  medical  simulation  is  a  viable  educational  tool  and  will  continue  to  proliferate 
(Wayne,  et  al,  2005).  Thus  research  into  a  standard  test  and  evaluation  methodology  is 
highly  desired. 
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Figure  1.  The  Stand  Alone  Patient  Simulator 
Stand  Alone  Patient  Simulation  (SAPS) 

A  revolutionary  technology  has  been  developed  in  the  research  and  development  phase 
for  medical  simulation,  the  Stand  Alone  Patient  Simulator  (Figure  1).  This  development 
represents  disruptive  technology  that  once  fielded  will  significantly  increase  combat 
medical  readiness.  The  SAPS  is  the  first  physiologically  based,  ruggedized,  wireless, 
feature  rich,  full  patient  simulator.  It  was  developed  under  the  Advanced  Medic  Training 
Technologies  -  Army  Technology  Objective  (AMTT-ATO),  to  address  specific  Army 
training  requirements,  and  to  demonstrate  the  application  of  advanced  technologies.  The 
SAPS  is  the  first  full-bodied  patient  simulator  designed  from  the  start  to  be  used  in 
simulated  combat  environments.  It  is  based  on  a  reinforced  skeletal  structure  with 
electronic,  hydraulic,  and  pneumatic  systems  integrated  under  a  foam  flesh  layer  covered 
with  realistic  skin.  The  SAPS  introduces  the  capability  to  train  critical  medical  skills  in 
rugged  terrain,  without  the  need  for  cumbersome  logistic  support.  The  SAPS  is  designed 
to  train  care  under  fire,  and  tactical  field  care  while  providing  the  capability  to  move  and 
treat  patients  through  all  levels  of  battlefield  medical  care.  It  is  also  capable  of  training  a 
majority  of  68 W  (Combat  Medic)  and  Combat  Life  Saver  (CLS)  critical  tasks  for  initial 
entry,  refresher,  transition,  and  sustainment  training,  thus  contributing  to  the  Army’s 
Simulation,  Training,  and  Instrumentation  mission  and  Army  Medical  Command’s  high 
priority  interests.  In  addition,  because  of  its  high  fidelity,  it  can  be  used  for  advanced  skill 
training  to  include  nurses  and  physicians,  as  well  as,  critical  thinking  and  team 
interaction.  It  provides  medical  educators  and  instructors  with  a  capability  to  objectively 
evaluate  student  performance  through  the  implementation  of  valid,  accurate,  and 
repeatable  medical  conditions  and  procedures  that  eliminate  subjective  influences  on 
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simulation  outcomes  or  assessment  of  student  behavior.  This  allows  it  to  be  used  both  for 
proficiency  and  mastery-based  training.  While  SAPS  is  partially  documented  in  a  test- 
model-test  approach,  additional  effort  must  be  dedicated  to  independent  test  and 
evaluation  before  it  can  be  fully  fielded  to  ensure  that  the  government  maximizes  all 
technological  inserts  available  and  establishes  the  suitability  of  use  for  each  level  of 
technology  demonstrated.  In  addition,  the  future  specifications  and  requirements  for  the 
formalized  acquisition  process  must  be  independently  established  and  completely 
deseribed  by  the  government  to  faeilitate  the  suitability  and  reliability  of  the 
proeurement,  production,  fielding,  and  life  eycle  support  effort.  This  project  will  execute 
an  expeditious  best  value  solution  to  the  documentation  requirement  for  the  research  and 
development  effort,  and  will  furnish  the  necessary  requirements  data  packages  to 
transition  the  SAPS  program  to  full  productization  and  acquisition. 
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Figure  2.  SAPS  Test  Design 

Design  Of  The  Effort  And  The  Test  And  Evaluation  Master  Plan 

The  SAPS  has  been  selected  under  the  DoD  Challenge  Program  to  be  accelerated  for 
production  and  fielding.  This  program  is  procuring  eight  SAPS  systems  for  both 
developer  design  improvement  and  independent  testing.  As  part  of  that  effort,  RDECOM 
has  instituted  an  Independent  Test  and  Evaluation  project  to  provide  complete 
documentation  to  move  the  SAPS  research  concept  into  the  acquisition,  production, 
fielding,  and  life  cycle  support  arena.  The  efficacy  of  this  project  has  been  reinforced  by 
DoD  Test  and  Evaluation  changes  in  a  December  22,  2007  memorandum  that  describes 
the  necessity  of  early  identification  of  technical,  operational,  and  system  deficiencies,  so 
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that  appropriate  and  timely  eorrective  actions  can  be  taken  prior  to  fielding.  The  Test  and 
Evaluation  Master  Plan  (TEMP)  has  been  constructed  using  DoD  Directive  5000.1  with  a 
modified  Naval  Air  Warfare  Center  Training  Support  Division  Trainer  Test  and 
Evaluation  Master  Plan  format.  Based  on  these  elements,  the  project  design  was 
constructed  with  two  integrated  phases.  Phase  One  is  to  establish  Critical  Operational 
Issues  and  Criteria  (COIC)  and  Measures  Of  Performance  (MOP)  and  Effectiveness 
(MOE)  for  the  SAPS  system,  create  a  Test  and  Evaluation  Master  Plan,  execute 
verification  assessment  of  the  research  effort,  conduct  independent  validation  test  and 
evaluation  of  the  SAPS  system  at  three  selected  joint  operational  medical  training  sites, 
and  fuse  and  analyze  the  data  collected.  The  processed  data  will  be  used  to  fully 
document  user  requirements,  to  verify  the  SAPS  delivery  to  the  government,  to  validate 
the  SAPS  concept  as  a  viable  instructional  tool,  and  to  prepare  accreditation  packages  for 
the  appropriate  agencies  for  specific  utilization  approval.  The  data  will  also  be  available 
for  full  peer  review.  Cost  will  be  considered  as  an  independent  variable  and  project 
progress  and  quality  of  effort  will  be  measured.  Consequently  the  project  will  produce 
inspectable  test  and  evaluation  and  assessment  data  and  system  engineering  functional, 
component,  and  data  flow  traceability  to  be  used  for  requirements  definition  (Eigure  2). 
Independently  testing  and  evaluating  the  SAPS  in  actual  use  environments  will  determine 
its  true  effectiveness  for  these  environments,  identify  weaknesses  in  the  design,  as  well  as 
in  training  and  help  provide  a  better  product  for  incorporation  into  the  DoD  medical 
training  centers. 

Test  Methodology 

The  first  phase  of  the  project  is  the  independent  test  and  evaluation.  The  test 
methodology  for  verification  will  be  to  use  a  prototype  simulator  with  a  go/no  go 
evaluation  of  thirty  designated  performance  items.  These  performance  items  will  then  be 
ratio  scale  graded  as  to  the  fidelity  of  their  implementation,  i.e.  low  fidelity  is  scaled  as  a 
2,  mid  fidelity  is  scaled  as  a  4,  and  high  fidelity  is  scaled  as  a  6.  Thus  an  item  rated  as  a 
low  fidelity  item  will  receive  a  composite  score  of  between  -2  to  +2  or  a  range  of  five. 
Eow  or  high  fidelity  is  not  “good”  or  “bad”  in  of  itself,  but  only  as  it  applies  to 
educational  effectiveness.  Some  medical  tasks  require  more  fidelity  than  others. 

The  test  methodology  for  assessment  of  performance  will  be  by  training  manager  and 
instructor  questionnaire,  observer/controller  grading,  and  by  instructor  evaluation  and 
student  pre  and  post-test.  No  control  group  will  be  used  since  each  of  the  test  locations  all 
use  patient  simulation,  and  patient  simulation  effectiveness  itself  is  not  test  item.  The 
efficacy  of  the  simulator  is  not  the  question.  It  is  the  appropriate  application  of  the 
developed  technology  to  the  right  Programs  Of  Instruction  (POI).  Instrument  validity 
will  be  conducted  prior  to  test  conduct  using  standard  validation  methods.  The  test  data 
will  also  be  verified  using  standard  statistical  correlation  methods. 

The  first  fielding  effort  will  most  likely  be  for  the  United  States  Army  Medical 
Simulation  Training  Centers  (MSTC).  The  MSTCs  are  currently  developing  an 
automated  training  effectiveness  measurement  system  for  medical  technical,  applied,  and 
tactical  skills.  Since  these  facilities  have  seen  the  most  use  of  patient  simulation  in  the 
military,  their  program  efforts  for  measurement  will  be  leveraged  within  this  effort, 
particularly  the  use  of  an  automated  measurement  tool  called  SIMIEE^"^  (Eigure  3). 
SIMILE™  is  a  government  development  co-jointly  sponsored  by  the  United  States  Navy 
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and  the  Advanced  Distributed  Learning  (ADL)  Co-lab.  The  primary  objective  was  to 
develop  a  middleware  application  to  integrate  simulations  and  games  into  a  SCORM 
compliant  Learning  Management  System  (LMS). 


Figure  3.  SIMILE™ 

The  SIMILE™  system  consists  of  a  user-friendly  graphical  interface  for  instructional 
designers  and  training  managers,  which  minimizes  pre-requisite  technical  knowledge  for 
simulation  or  game  data  schemas.  It  has  automated  after- action  review  organizational 
capabilities  hereafter  referred  to  as  performance  assessment  routines,  and  a  feedback 
engine  that  provides  evaluation  of  student  performance  and  instructional  methodology. 
This  makes  it  an  ideal  candidate  for  test  and  evaluation  data  management. 

The  entire  effort  will  furnish  independent  test  and  evaluation  data  to  establish  the 
verification,  validation,  and  potential  accreditation  for  the  SAPS.  In  addition,  the 
application  of  system  engineering  processes  for  requirements  traceability  will  ensure  that 
the  test  data  is  formatted  for  use  in  the  acquisition  cycle. 

System  Engineering 

The  second  phase  of  the  project  is  to  conduct  detailed  system  engineering  processes  to 
create  the  requirements  and  specifications  for  a  production  level  product.  This  will 
include  a  detailed  task  analysis,  a  fully  documented  functional,  component,  and  data  flow 
design,  and  a  productization  process  schedule.  This  effort  will  be  based  directly  on  the 
independent  test  and  evaluation  data  gathered  in  the  first  phase  with  full  traceability  to 
the  entire  SAPS  research  and  development  effort.  The  resulting  package  will  be 
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formatted  in  an  acquisition,  production,  and  fielding  template  to  facilitate  the 
procurement  process  for  the  next  generation  of  military  patient  simulators. 

For  SAPS,  government  sponsored  user  tests  have  demonstrated  three  distinct  needs; 
break  away  extremities,  increased  durability,  and  lowered  cost.  The  three  critical  needs 
identified  above  will  be  independently  evaluated.  Also,  the  development  Program 
Manager  wants  an  independent  assessment  of  fidelity  to  educational  application  to  ensure 
requirement  suitability. 

Critical  Operational  Issues  And  Criteria 

A  detailed  analysis  of  the  current  educational  literature  with  particular  attention  on  a 
variety  of  military  occupational  specialty  educational  requirements  was  conducted.  In 
essence,  those  tasks  all  fell  within  specific  domains  for  patient  assessment,  trauma 
management,  treatment,  and  evacuation.  In  addition,  these  tasks  were  executed  in  four 
specific  environments;  care  under  fire,  tactical  field  care,  casualty  evacuation,  and  high 
level  care  facilities.  These  tasks  were  correlated  to  the  SAPS  BAA  requirements.  In 
addition,  user  test  deficiencies  as  well  as  Program  Manager  concerns  were  combined  to 
create  seven  COIC. 

The  first  issue  was  the  simulator’s  stand  alone  capability.  This  issue  was  at  the  core  of  the 
research  and  development  effort.  Previously,  patient  simulators  had  been  limited  in  their 
mobility  because  they  were  tethered  to  numerous  devices;  gas  containers  to  make  them 
breathe,  electric  power  sources,  fluid  reservoirs  for  bleeding,  and  computer  control  units. 
These  tethers  made  field  use  and  realistic  casualty  evacuation  almost  impossible. 

The  second  issue  was  the  simulator’s  robustness  for  field  training  (particularly  joints  and 
skin).  During  user  tests  conducted  during  the  development  phase  there  was  a  high  degree 
of  unreliability,  particularly  with  the  joints  and  the  skin,  both  of  which  were  new 
developments.  Previously,  patient  simulators  had  insufficient  movement  capability  to  be 
realistically  evacuated  or  moved.  In  addition,  skin  was  highly  unrealistic  both  in 
appearance  and  behavior. 

The  next  three  items  were  the  simulator’s  anatomical  reality,  the  simulator’s  clinical 
accuracy,  and  the  simulator’s  quick  reconfigureability  for  lane  training. 

Finally,  the  last  two  items  were  particular  concerns  of  the  Program  Manager,  the 
simulators  cost,  and  the  applicable  fidelity  to  the  appropriate  POT 

Site  Selection  And  Population 

Four  sites  were  chosen  to  conduct  the  test,  one  for  verification  and  three  for  validation. 
The  verification  test  will  be  conducted  at  RDECOM  in  Orlando,  Florida  using  simulation 
and  military  clinical  subject  matter  experts.  Because  it  is  to  verify  traceability  of 
instantiated  requirements,  no  test  population  is  necessary.  The  validation  sites  were 
chosen  for  their  ability  to  support  the  test,  the  necessity  for  joint  service  evaluation,  the 
experience  of  their  instructors,  and  the  diversity  of  their  students.  The  sites  include  a  DoD 
Medical  Training  facility,  a  U.S.  Army  MSTC,  and  a  U.S.  Marines  Corps  facility,  which 
trains  Navy  corpsmen. 

The  DoD  facility  is  the  Defense  Medical  Readiness  Training  Institute  (DMRTI)  located 
in  San  Antonio,  Texas  and  was  chosen  because  of  its  mission,  instructors,  and  student 
base.  DMRTI  is  responsible  for  training  all  levels  of  students  from  physician  to  medic  in 
tactical  combat  casualty  care.  Its  instructors  are  some  of  the  most  experienced  in  DoD 
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and  most  are  combat  veterans.  They  have  a  tactically  realistic  simulation  center  with  full 
observational  control  and  data  recording  abilities,  and  have  SAPS  on  site.  Their 
command  willingly  supports  research  and  test  and  evaluation.  They  are  a  joint  facility 
that  trains  all  services. 

The  U.S.  Army’s  MSTC  located  at  Ft.  Bragg,  North  Carolina,  supports  the  XVIII 
Airborne  Corps,  whose  units  have  extensive  deployments  to  both  Iraq  and  Afghanistan.  It 
is  supported  by  both  military  and  contract  instructors.  The  students  range  from  Combat 
Life  Savers  (CLS),  who  are  non-medical  personnel,  and  68W  Soldier  Medics.  They 
conduct  skills,  lane  and  validation  (test)  training. 

The  training  facilities  for  U.S.  Navy  corpsmen  supporting  the  U.S.  Marine  Corps  are 
located  in  San  Diego,  California,  and  Camp  LeJune,  North  Carolina.  Their  instructors  are 
strong  advocates  for  patient  simulation.  Their  students  are  corpsmen  who  have  completed 
initial  entry  training  who  upon  completion  of  training  deploy  to  forward  U.S.  Marine 
Corps  units.  Their  training  is  combat  skills  based,  and  they  have  observational  control 
and  data  recording  capabilities. 

In  each  case,  on  site  training  managers  are  totally  involved  in  test  planning  and  conduct 
to  ensure  validity  of  results.  Site  command  receives  entrance  and  exit  briefings. 
Confidentiality  and  control  of  data  is  ensured,  and  neither  test  personnel  nor  procedures 
will  interfere  with  their  on-going  training  mission. 

Measures  Of  Performance  (MOP)  And  Measures  Of  Effectiveness  (MOE) 

MOPs  will  be  used  in  the  verification  portion  of  the  test.  MOEs  will  be  used  in  the 
validation  portion  of  the  test.  Both  will  support  the  accreditation  package  preparation.  All 
of  the  above  data  will  support  the  system  engineering  process  to  prepare  the  acquisition 
package. 

There  are  thirty  measures  of  performance  identified.  They  are  divided  into  four 
categories;  operating  requirements,  feature  requirements,  system  requirements,  and 
educational  requirements  (Figure  4).  The  measurement  criteria  are  four  rated  categories 
with  comment.  The  MOP  either  fails  to  meet  the  requirement,  partially  meets  the 
requirement,  meets  the  requirement,  or  exceeds  the  requirement. 

There  are  sixty-five  MOEs  which  are  based  on  Training  Circular,  TC  8-800,  United 
States  Army,  Medical  Education  And  Demonstration  Of  Individual  Competence,  dated 
January  2008,  primarily  Table  VIII  which  is  the  hands-on  skills  testing.  These  tasks  are 
the  primary  source  for  68W  training  and  are  grouped  into  six  categories;  trauma 
assessment  and  treatment,  airway  management,  intravenous  access,  medications  and 
management,  medical  assessment  and  treatment  to  include  Chemical,  Biological, 
Radiological,  and  Nuclear  (CBRN),  triage  and  evacuation,  and  cardiopulmonary 
resuscitation  management.  Table  VII,  obstetric,  gynecology,  and  pediatric  treatment  was 
excluded,  as  was  full  decontamination  for  CBRN.  The  measurement  of  effectiveness  will 
be  determined  by  student  cognitive  skills  pre  and  post-test,  coupled  with  patient  outcome 
as  determined  by  recorded  physiological  state,  both  correlated  to  the  instructor’s  rating  of 
task  performance.  The  test  will  be  controlled  by  subject  matter  expert  observational 
checklist. 
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MEASURES  OF  PERFORMANCE 


FEATURE  REQUIREMENTS 

1.  Physiologically  correct  reactions 

2.  Simulator  weight  should  replicate  soldier 

3.  Simulator  should  hend  as  a  human 

4.  Airway  anatomically  correct 

5.  Pulses  reactive  to  physiological  change 

6.  Realistic  pulse  rate  and  quality 

7.  Bleeding  and  fluid  loss 

8.  Blinking  eyes  and  reactive  pupils 

9.  Realistic  heart  sounds/hreath  sounds 

10.  Basic  psychomotor  skills  for  trauma  tres 

11.  Simulate  orthopedic  injuries 

12.  Flail  chest 

13.  C-spine  immohilization 

14.  Tremors 

15.  Multiple  drugs 

16.  Surrogate  drug  injection  devices 

17.  Specific  drugs 

18.  Anatomical  reality 


OPERATING  REQUIREMENTS 

1.  Free  of  all  external  connections 

2.  Internally  powered  w/external  charge 

3.  External  power  source  llOv  with 
options 

4.  No  scenario  interruption  w/power 
change 

5.  Wirless  central  control 

6.  Capability  to  introduce  complications 

7.  Field  ruggedized 

SYSTEM  REQUIREMENTS 

1.  Documented  network  interface 

2.  Multiple  simulators  centrally 
controlled 

3.  Networkahility  and  interoperahility 

4.  Cost  of  $30K 

EDUCATIONAL  REQUIREMENTS 

1.  Support  all  68w  tasks  with  two 
exceptions 


Figure  4.  Measures  Of  Performance 
Test  Conduct  And  Test  Data  Management 

The  verification  test  will  be  conducted  using  a  relational  matrix,  which  is  a  yes/no  (go/no 
go)  test.  The  research  prototype  either  does  the  function  or  does  not.  There  will  be  a 
correlated  column  data  item  to  rate  the  fidelity  of  the  tested  function  by  the  Subject 
Matter  Expert  (SME),  with  three  categories;  high  degree  fidelity,  medium  degree  fidelity, 
low  degree  fidelity.  The  fidelity  rating  will  be  scale  graded  based  on  the  functional 
performance  requirements.  In  addition,  the  specified  requirements  will  be  directly  traced 
to  the  educational  tasks.  Retesting  of  any  identified  discrepancies  will  be  determined  by 
the  government. 

The  validation  tests  will  be  conducted  at  the  above  identified  training  locations. 
Questionnaires  will  be  distributed  to  training  managers,  instructors,  and  students  to 
determine  the  efficacy  of  each  instantiated  function.  Selected  scenario  events  will  be 
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observed  and  reeorded  to  reinforee  the  questionnaire  data.  Student  performanee  will  be 
observed  for  elinieal  aecuraey  and  rated  based  on  patient  outeome.  Learning  will  be 
evaluated  by  pre  and  post-test  of  task  eognitive  skills.  Trainer  downtime  limitations 
during  testing  will  be  identified.  Testing  will  not  be  terminated  due  to  identified 
defieieney  reports  or  inoperative  trainer  systems  and  subsystems. 

The  aecreditation  paekages  will  be  based  on  validation  testing  results  and  instructor  input 
for  selected  Programs  Of  Instruction  at  each  location.  Training  efficacy  assessment  will 
be  done  by  a  computer  based  program,  SIMILE™,  to  ensure  objectivity. 

The  Application  Of  The  Results  To  V,V&A 

In  the  conduct  of  this  test,  the  following  definitions  were  used  to  describe  V,V&A  (Elele, 
2008).  Verification  is  the  process  of  determining  that  a  model  implementation  and  its 
associated  data  accurately  represent  the  developer’s  conceptual  description  and 
specifications.  The  practical  question  answered  by  verification  is  “Is  the  model  relatively 
error  free,  and  does  it  do  what  the  originator  intended?” 

Validation  is  the  process  of  determining  the  degree  to  which  a  model  and  its  associated 
data  are  an  accurate  representation  of  the  real  world  from  the  perspective  of  the  intended 
uses  of  the  model.  Note  that  M&S  validation  is  not  the  same  as  software  validation.  The 
practical  question  answered  by  validation  is  “Do  model  results  match  real  world  data  well 
enough  for  the  user’s  needs?” 

Accreditation  is  the  official  determination  and  certification  that  a  model,  simulation,  or  a 
federation  of  models  and  simulations  and  its  associated  data  are  acceptable  for  use  for  a 
specific  purpose.  The  practical  questions  answered  by  accreditation  are  “Does  the 
accreditation  authority  have  adequate  documented  evidence  to  be  confident  that  a  model 
and  its  input  data  are  credible  for  a  particular  use;  and  is  that  enough  documented 
information  to  show  that  this  M&S  is  fit  for  this  purpose?” 

These  definitions  are  composites  that  fit  the  joint  nature  of  the  test. 

The  verification  test  was  conducted  in  January  2009.  The  preliminary  results  are  reported 
below  as  well  as  the  lessons  learned.  The  verification  test  was  conducted  in  Orlando, 
Elorida  independently  of  any  student  or  instructor  presence.  It  was  different  than  standard 
verification  in  that  in  addition  to  rating  requirement  delivery,  it  also  assessed  requirement 
traceability  to  educational  standards.  Past  verification  tests  in  most  cases  assumed  valid 
requirements. 

The  validation  tests  will  be  conducted  in  the  spring  of  2009.  Eor  medical  simulation, 
validation  must  be  based  ultimately  on  patient  outcome.  Thus  physiological  states  must 
be  known  and  accurate.  Also  for  medical  simulation,  the  reflection  of  procedural  or 
algorim  accuracy  is  a  key  measurement  to  ensure  that  the  physiological  state  achieved  is 
directly  related  to  the  clinical  intervention.  Additionally  for  medical  education  cognitive 
task  expression  is  a  measure  of  confidence  level,  a  critical  performance  determinant  as 
well  as  a  measure  of  educational  retention. 

The  accreditation  packages  will  be  completed  and  submitted  to  the  various  agencies  in 
the  summer  of  2009.  In  essence,  these  will  indicate  by  test  data  analysis  what  particular 
instructional  items  have  been  fully  validated  with  an  accreditation  recommendation.  This 
does  not  constitute  a  product  endorsement. 
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Test  Results 

To  date  the  only  reported  results  are  preliminary  for  the  verifieation  test  condueted  in 
January  2009.  Of  the  MOP,  the  SAPS  fully  met  six  of  the  seven  operating  system 
requirements  and  partially  met  the  seventh,  field  ruggedness.  The  issue  was  the  strength 
of  the  simulated  skeletal  joints. 

The  SAPS  fully  met  thirteen  of  the  feature  requirements,  partially  met  three,  and 
exeeeded  one.  There  was  only  one  feature  that  was  not  met,  replieation  of  orthopedie 
injuries. 

Of  the  system  requirements  three  were  fully  met  and  one  was  not.  The  one  not  met  was 
the  cost. 

Of  the  education  task  requirements,  sixty  of  the  sixty-five  TC8-800  tasks  could  be 
performed  on  the  SAPS. 

The  level  of  fidelity  is  still  being  evaluated,  as  well  as,  the  educational  needs  to 
requirements  traceability. 

Lessons  Learned 

Though  the  test  results  are  preliminary,  four  initial  lessons  learned  have  emerged. 

The  first  is  that  any  education  or  training  simulation  research  development  must  be  firmly 
grounded  in  educational  requirements  and  that  these  requirements  must  be  specified. 
Three  of  the  five  educational  requirements  not  met  were  not  specified.  The  other  two  not 
met  were  based  on  the  failure  to  meet  one  feature  requirement. 

The  second  lesson  learned  is  that  effective  developmental  testing  is  essential  to  eliminate 
failure  and  to  provide  risk  mitigation  in  assuring  accurate  delivery.  The  above  failure  was 
not  identified  in  the  initial  user  test  conducted,  indicating  a  developmental  test  failure. 

The  third  lesson  learned  is  that  high  fidelity  should  not  be  included  for  its  own  sake. 
Missing  the  system  objective  of  cost  has  significant  implications  for  the  acquisition 
package.  The  suspected  cause  of  this  failure  was  an  over  aggressive  application  of 
unnecessary  fidelity  and  technology.  This  will  be  verified  when  the  fidelity  data  and 
validation  data  is  compiled.  Again,  the  focus  has  to  be  on  meeting  the  specific 
educational  requirements  with  precisely  specified  performance  parameters.  The  fourth 
lesson  learned  is  that  in  program  requirements  ruggedization  must  be  expressed  in 
reliability,  availability,  and  maintainability  measurements.  Field  ruggedness  presented 
significant  problems  during  the  development.  While  field  ruggedness  was  specified  in  the 
initial  requirements,  it  was  never  fully  defined. 


Conclusion 

The  independent  test  and  evaluation  process,  when  applied  with  V,V&A  principles  and 
directly  tied  to  system  engineering  processes,  serves  as  an  important  risk  mitigator  to 
ensure  program  success.  In  addition,  appropriate  application  of  this  process  has  the 
potential  to  produce  significant  cost  savings  for  acquisition  and  life  cycle  costs. 
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